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Abstract 


In October of 2019, I presented at the IUPUI Assessment Institute in Indianapolis on the topic of academic program review. 
While almost every campus conducts program reviews, the definition, purpose, structure, and integration of assessment 
in program reviews is not universally understood, accepted, or implemented. This paper summarizes essential questions 
and discussions campuses should have regarding program reviews and the integration of assessment into such reviews. 
Unfortunately, too many campuses simply keep doing the same reviews over and over. The goal for this paper is to cause 
campuses to pause, reflect, answer some questions, and then revise their reviews to ensure a meaningful product. The paper 
is organized around the essential topics of definitions, accreditation, processes, continuous improvement and “closing the 
loop,” and potential challenges. This paper concludes with templates from McKendree University that may serve as valuable 
tools which other campuses may use, edit, or compare to their own processes and practices. 
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Program Review and Assessment for Continuous ~ 


Improvement: Asking the Right Questions 


Tami Eggleston 
McKendree University 


At many campuses, we do things because we have always done them that way. Change 
is hard in academia and once we put some process, curriculum, program, template, or 
project into motion it typically continues over time. In academia, processes very much 
follow classic physics arguments that an object in motion tends to stay in motion. Many 
institutions are great at adding new programs, new reports, and new activities but not 
great at taking time to reflect and even eliminate items that are not working. But over 
time, we lose sight of what we are talking about and working towards, or why these 
processes and practices were even put into place. While almost every campus conducts 
program reviews, the definition, purpose, structure, and integration of assessment in 
program reviews is not universally understood, accepted, or implemented. This paper 
summarizes essential questions and discussions campuses should have regarding program 
reviews and the integration of assessment into such reviews. A good starting place is to 
begin with developing shared definitions. 


Question and Discussion #1: 
What are We Even Talking About? Developing Shared Definitions 


Program Review: Program review is a comprehensive look at individual programs and 
covers staffing, curriculum, learning outcomes, action plans, etc. They often involve an 
external reviewer, and usually at least give lip-service to the importance of assessment. 
Program reviews are required and/or strongly suggested by most accrediting bodies (to 
be further discussed below). Program reviews typically have at least a guiding outline or 
template for items to be included or reviewed. Most program reviews are written by the 
faculty members within that program and therefore are heavily geared toward curriculum, 
staffing, and resource needs. How often program reviews are completed is debatable and 
discussed below. For a sample of a program review, please see Appendices associated with 


this paper. 


Program Assessment Reports: Assessment reports focus exclusively on assessment. They 
focus on outcomes, student learning, assessments, and closing the loop. Not all campuses 
or all divisions or schools engage with these specific program assessment reports. These are 
more focused and would allow time spent just on assessment rather than on curriculum, 
staffing, financials, etc. Theoretically these could be completed every year or every other 
year because assessment is ongoing and continuous improvement is important. For a 
sample of a program assessment report, please see Appendices associated with this paper. 


Program Prioritization: With a program prioritization process, an institution would 
8 

put ALL programs (academic, athletic, student affairs, etc.) on a chart to compare and 

prioritize (often involving financial information). Program prioritization involves: 


* Comparison of programs against other programs (e.g., what major has more 
students). 

¢ Deciding what programs should be improved, what programs should be 
developed, and what programs should be eliminated. 


Program review is a 
comprehensive look at 
individual programs that 
covers staffing, curriculum, 
learning outcomes, action 
plans, etc. They often 
involve an external reviewer, 


and usually highlight the 


importance of assessment. 
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¢ Less emphasis on student learning and more emphasis on numbers, financials, 
and a propensity for success in the future. 


A program with one student could have excellent student learning outcomes, 100% 
retention, and a 100% graduate school placement rate with an amazing assessment 
program (I would imagine a program with one student could have an excellent, detailed 
portfolio with pre/post measures). But that program with just one or a few students is 
likely not financially viable over time for most institutions. 


Program prioritization takes a village to complete with faculty, staff, and administration 
across the campus involved, and takes time to gather data. It is usually undertaken due 
to financial concerns, a new mission, or new leadership. Campuses usually complete a 
program prioritization and then do not do another for a few years. Although program 
prioritization is necessary and important for many campuses, it is less likely to concern 
itself with assessment and therefore is not the topic of this paper. Faculty are rightfully 
not excited about program prioritization and therefore it may be a best practice (if not 
a popular practice) to have program prioritization completed on a timeframe such as 
every seven to ten years so that it seems less like a “crisis” when it happens and more like 
business as usual.! 


Shared Definitions: Some campuses may do all three of these processes, some campuses 
only one of these activities, and some campuses may alternate or vary these processes. It 
is essential that institutions know what they do and when they do them and call them 
by the correct name. Calling a program review, a program prioritization is not correct 
and could create a false sense of alarm. And calling a program prioritization a program 
review, is not correct and could create a lack of transparency and honesty about the 
desired information and outcome. To be a bit controversial, (because what good is a paper 
without some controversy?), an administrator may think a campus is doing prioritization 
when it is only doing review and a faculty member may think they are doing program 
review when in fact it is a prioritization. During my research for this paper, I found a 
very interesting online report by Hanover (2012) that has some fascinating theories and 
strategies about best practices in program review. They call this report a program review, 
but I would argue it is most definitely a program prioritization process. The point is that 
your campus community needs to decide for yourself what the different elements are and 
are not. 


Assessment: The purpose of this paper is to explore how assessment can be included in 
program reviews. If the institution decides that assessment does not fit into program 
review, then there needs to be articulation of the goals of the program reviews with 
program assessment reports occurring on a different time frame. 


Just to be clear, let’s briefly define assessment. One issue with assessment is the multitude 
of definitions and examples. Psychologists and counselors use assessment to describe the 
initial activities for examining an individual to determine concerns or issues. Assessment 
can also be used strictly in the classroom for activities such as tests and papers. Assessments 
in the classroom lead many faculty to think, “Well of course I do assessment, I give 
tests and a final paper.” And then many accreditors (who should know better!) use the 


'For an excellent book on program prioritization, please see Dickeson, R.C. (2010). Prioritizing academic 
programs and services: Reallocating resources to achieve dtrategic balance. Hoboken, NJ: John Wiley and Sons. 
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language of assessment data/outcome data when they really do not mean student 
learning outcomes, and instead mean outcomes such as retention, persistence, and 
graduation rates. It is not surprising that we have difficulty with our assessment efforts 
with so many different denotations and connotations floating around. When 
assessment experts (or those of us who actually read the books, attend the conferences, 
and truly care about this work) define assessment, we usually mean student learning 
outcomes and the associated process of ensuring we are meeting the benchmarks or 
objectives we stated. 


One thing most experts agree upon is that assessment is more than a grade. A 
student can earn a “C” in a class, but that tells us little about why they earned the “C.” 
Did they achieve the learning outcomes and truly reach the level we wanted on some 
construct (e.g., critical thinking) but simply not attend class or not do well on low-level 
knowledge-based exams? Or did the student earn a “C” because they did all the 
work, were there every day, and memorized information to pass the exams, but 
actually did not exhibit critical thinking on the final paper or project? These are two 
very different reasons for the “C” and raise confusion on student learning. So, to put 
it simply, grades are a good measure for grading but not a good measure for student 
learning outcomes.” 


To define assessment just a bit more before we get into the role of assessment in 
program review, assessment can take place at three levels: 


¢ The course level, what happens in individual classes (e.g., a paper, an exam, 
etc.) 

¢ ‘The program level, what happens in a program (e.g., aggregated assignments, 
pre/post tests, graduate school placement, etc.). 

¢ And the institutional level, what happens across the institution (e.g., rubrics 
attached to assignments, surveys, etc.). 


To further muddy the waters, one item or project could be assessed and aggregated at the 
course level, information from which could be used for the program level, and finally it 
could be used for the institutional level as well or the co-curricular.’ 


Question and Discussion #2: 
What do Your Accreditation Organizations Have to Say? Aligning with External 
Accountability Requirements. 


Every program on a campus should do a program review to ensure they are meeting 
their stated student learning outcomes. Many programs (e.g. nursing, business, athletic 
training, education) are externally accredited and typically have a history of assessment 
activities because it is required. While assessment is for much more than accreditation 
purposes, it is important to ground conversations in what your accrediting organization 
says about assessment, outcomes, student learning, and specific criteria or assumed 
practices they require. 


*For more on this topic, see Suskie, L. (2018). Assessing student learning: A common sense guide. Hoboken, NJ: 
John Wiley and Sons; and Suskie, L. (2015). Five dimensions of quality: A common sense guide to accreditation 
and accountability. San Francisco, CA: Jossey-Bass. And for resources on a variety of assessment activities 


visit the NILOA webpage: https://www.learningoutcomesassessment.org 


3For more information about levels of assessment, please visit https://www.aacu.org/sites/default/files/files/ 


We should engage in 
assessment at institutions 
because we care about student 
learning and continuous 
improvement. But we also 
should clearly know what we 
are required to prove in terms 
of this work to our accreditors 
(at the both the program and 
institutional levels). 
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I want to emphasize, that we should engage in assessment at institutions because we 
care about student learning and continuous improvement. But we also should 
clearly know what we are required to prove in terms of this work to our accreditors 
(at both the program and institutional levels). For some reluctant faculty and staff, 
using language from accreditors may be helpful to establish external validity. When 
faculty or staff ask me why accreditors make “such a big deal about assessment?,” I 
respond that the accreditors understand the value that the assessment process adds to 
campuses. Having these discussions allow us to ensure we are providing a high-quality 
educational experience to students and providing evidence that we do what we say we 
do and that we do it well. 


For example, the Higher Learning Commission requires, under core component four, that 
“The institution demonstrates responsibility for the quality of its educational programs” 
and “The institution maintains a practice of regular program reviews.” What the HLC 
does not say is what “regular” means or what must be included in the program reviews - 
that is up to the institution to determine and decide. 


SACSCOC states that “The institution identifies expected outcomes, assesses the extent 
to which it achieves these outcomes, and provides evidence of seeking improvement 
based on analysis of the results in the areas below: a.) Student learning outcomes for each 
of its educational programs, b.) Student learning outcomes for collegiate-level general 
education competencies of its undergraduate degree programs, and c.) Academic and 
student services that support student success.” 


Most accrediting bodies will lay out the criteria and provide some best practices, but allow 
institutions flexibility in the exact who, what, where, when, and how of program review. 
Of course, start with the accreditation criteria, but make it work for your campus. The 
purpose of assessment is not just for accreditation or just to improve your campus, but 
truly to improve higher education (Kuh et al., 2015). 


Question and Discussion #3: 
What Will Work for Your Campus: The Why, When, How, Who, and What of 


Program Review 


WHY? Spend some time with your campus deciding on the why. Why are you doing 
each of these activities? What is the ultimate outcome, why are you doing them? Skipping 
this step will result in less faculty and staff interest and engagement. And the why should 
include more than just accreditation compliance. 


WHEN? The next q uestion b ecomes when. I have consulted at campuses t rying to 
simply do too much and conducting assessment every year with little time to reflect, 
make decisions, try to influence change, and see if the change worked. At McKendree, we 


used to have assessment reports due every year, but we found this was simply too much. 
We now have the assessment reports due every other year (on the odd years) and a very 
brief action plan about what needs to be done every other year (on the even years). For 
our assessment reports, we fear if we go any longer that assessment may go on the back 
burner and be ignored which is problematic because good assessment requires constant 
attention. For our program reviews because of the detail of our review, as well as the use of 
external reviewers and the amount of time it takes to change curriculum and assess these 
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changes, we have our programs on seven-year cycles. Seven years may seem like a long- 
time at first, but doing a program review with quality, with intention, and with time for 
reflection is likely better than doing program reviews more often with less quality. Quality 


is usually better than quantity when it comes to assessment initiatives.‘ 


HOW and WHO? After definitions and the big picture and timeline are decided, a clear 
process must be detailed for any activity to be successful. At McKendree we start with 


a self-study, then an external review committee, and then an action plan. A department (QQ 


will spend almost two years on the entire process. An important part of the how also WUNEOcUIn anda 


they do not like assessment 

or do not see the value in 
assessment, I think it is largely 
because we do not pause to: 

1. think about the purpose; 
2. ask good questions; and 

3. be truly willing to change 


becomes who should be on the committee(s), who should receive the reports, and who 
will provide feedback? There should be some form of closing the loop with any report, 
such that there is feedback and an action response. If feedback or reflection is not built 
into the process, it likely will not happen and will make the process less valuable. One 
of the last, but important, questions with the how are the logistics of where they will be 
submitted, how they will be housed, how long they will be retained, who has access to 
viewing the reports, etc. 


curriculum, course 
offerings, assignments, 
etc. for the good of the 


student. 


WHAT? Another important part of this “How” quickly becomes the what. For both our 
assessment reports and our program review, we have a clear and simple template (See 
related Appendices). It is important to regularly look at this template and ensure that 
every part of the required template is valuable and used. Also included in the related 
Appendices for this paper is a rubric the committee uses to evaluate the quality of the 
assessment report. For the program review, we use a narrative approach with external 
reviewers and committee members providing feedback. Additional Appendices include the 
complete McKendree University Program Review Process and links to other institutions 
with clearly defined program review processes. Also, readers will find a slightly edited 
completed psychology program review that may be helpful for workshops, training, or 
ideas.” 


Question and Discussion #4: 
So What? How Will We “Close the Loop” & Engage in Continuous Improvement? 


After all of this work to make a process, create a report, and review the report, it is 
essential that campuses use the results gleaned from any of these reports. There should be 
opportunities for reflection and sharing built into the process. In addition to reflection, 
at McKendree University we have a “Closing the Loop” all-day workshop each May 
where we share data from the institution and/or the programs to support professional 
development to address areas identified in program review. This is an opportunity for 
workshops on a variety of topics such as curriculum maps, best practices in assessment, 
samples of assessment activities, results from surveys, etc. We also have teaching workshops 
each semester that focus on topics such as using writing rubrics to improve assignments, 
adding critical thinking assignments, ways to add diversity discussions, etc.° 


“For more information about program reviews and timelines, please visit our McKendree University 
webpage https://www.mckendree.edu/offices/ provost/academic-program-review.php 


>For a detailed summary of program reviews, please see Bresciani, M.J. (2006). Outcomes-based academic 
co-curricular program review. Sterling, VA: Stylus Publishing. 


“For more information on a useful resource for teaching workshops and encouraging best practices in 


assessment activities at the course and program level, please see Richmond, A.S, Boysen, G.A, and Gurung, 
RALR. (2016). An evidence based guide to college and university teaching: Developing the model teacher. New 
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In recent years, we have been more intentional with adding student affairs to our 
assessment activities and in our teaching workshops. The issue of assessment, program 
reviews and student affairs is worthy of a separate white paper, but for a good overview of 
program review in student affairs please visit: Council for the Advancement of Standards 
Program Review Standards for Student Affairs https://www.cas.edu/programreview or see 


Henning, Robbins, and Andes (2020) paper. 


When we keep the focus on student learning and continuous improvement, the program 
review process evolves to ensure that we are getting the information we need. Essentially 
a process of meta-assessment starts to occur when you assess your assessment efforts. 


When faculty and staff say they do not like assessment or do not see the value in assessment, 
I think it is largely because we do not pause to: 1) think about the purpose; 2) ask good 
questions; and 3) be truly willing to change curriculum, course offerings, assignments, 
etc. for the good of the student. Many times, when I work with faculty at various 
campuses, the faculty are more concerned about the courses they want to teach and the 
assignments they want to do, as opposed to being concerned with student learning—a 
shared responsibility between students and faculty. It takes a great deal of reflection and 
willingness to change for assessment to truly work and lead to continuous improvement. 


Question and Discussion #5: 
Forewarned is Forearmed: What are the Potential Problems? 


For the last ten years I have reviewed and consulted with between 50 and 100 institutions. 
And as much as I believe in the power of assessment, belief in the process of assessment 
is not universal. We have a problem and part of that problem has been bad assessment 
processes that start when you do not begin with questions and design the right processes 
from the start. My psychology background has been immensely helpful as I try to 
understand the resistance to assessment across faculty and staff. Reasons can range from 
simple laziness (who wants more work?) to an overarching distrust of administration and 
all things that come from the Provost’s office, to more valid reasons such as “we do all of 
this work for nothing”. Below are my top 11 (tongue-in-cheek) issues and challenges. 


#11. The “Maybe assessment will just go away” hope. 

Let’s face it, in academia things come and go. We were all excited about MOOCs a few 
years ago, and then we all ‘flipped’ over the flipped classroom, and let’s not forget clickers. 
So yes, the idea that assessment is just another trend could have been true 30 years ago, 
but it is here to stay. But hey, maybe the new Provost won't care about it! Further, a 
significant book about assessing academic programs came out in 2004, so let’s be clear, 
assessment in programs is nothing new (Allen, 2004). 


#10. The “Just get it done!” check officheck out. 

You get the memo, we have to do a program review. Let’s not think, reflect, or get the 
program together. We are all busy faculty with teaching, advising, service, research, let’s 
just assign it to someone to get it done. Not doing a good job just means you won't be 
asked to be on the committee! 
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#9. The not “Closing the loop” loophole 

You do the program review, turn it into the Provost in May and you never look at again 
until the next time it is due. The data may be collected, but it is never used, it is never 
reflected upon, and nothing is ever done to improve learning. However, you are very 
good about submitting a report on time! If we don’t use the data and close the loop, then 
assessment really isn’t worth it. If assessment falls in the forest and no one is there to use 
the data, did it make a noise? 


‘The best part of a program 
review is the program getting 
together and defining goals, 
talking about courses and 
talking about what matters, 
so if just one person (e.g., the 


#7. The “Lost in the dean’s office” blackhole chair) does it, that defeats the 
Part of the reason that faculty may not care about assessment or program reports is ifthey entire purpose. 

submit them and never receive any feedback. If programs are not rewarded (or punished) 

and it just seems like reports go into the blackhole along with surveys, the NSSE reports, 

and all other things we collect, why should we care? How many wonderful, useful surveys 


and results are just sitting dusty on a desk somewhere or stashed in a file cabinet or lost 
in a shared folder? 


#8. The “Turn in last year’s report!” scheme 

If we don't get rewarded or punished for our program reviews or assessment reports, no 
one is looking at these things anyway, just turn in the report from last year. Even worse, I 
have heard that faculty have plagiarized other people’s reports internally or externally! The 
same faculty that would be horrified if a student turned in the same paper twice, don’t see 
any harm doing it themselves. 


#6. The “That’s the chair’s job” mindset 

The best part of a program review is the program getting together and defining goals, 
talking about courses and talking about what matters, so if just one person (e.g., the 
chair) does it, that defeats the entire purpose. Part of assessment is not just the outcome or 
end result, it is the messy, democratic, discussion process. Allow time for this to happen 
(but not too much time). 


#5, The “Who made this template?” distraction 

It is essential to have a template to help guide people and to help standardize the process. 
But complaining about the template can be a distraction technique. Have a set timeline 
for when, where, and who can change the templates occasionally. This is closely related to 
“who made this rubric.” It is so much more fun to just complain about a rubric than to 
actually use it and try to improve student learning. 


#4, The “Anything goes” no template game plan 

Some argue that every department and program is unique so no template can capture 
what they are doing. Even if you have a template or rubric that needs work, that is better 
than having the Education department turn in a 290-page report and the mathematics 
department turn in a five-page Excel spreadsheet. You need a template for the report, and 
you need a template to provide feedback. These templates should be reviewed on a cycle 
(and that cycle is not every year!). 


#3. The “Next year we will address it...” procrastinator’s promise 
Change in academia is slow, academics like to say they need more information, they need 
to look at other schools, they need to revise the rubric, they need to design a new course, 
they promise to do it tomorrow—and tomorrow never comes! A timeline needs enough 
time to do quality work, but a short enough time to get work done. 
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#2. The “What are they going to do to me if I don't” dilemma 

In academia we rely on the intrinsic motivation and good faith of most people. But 
sometimes people (faculty, staff, and administrators) don’t want to do this work. They 
may be busy with other important tasks or simply don’t prioritize this work. It is hard to 
punish or reward assessment activities, but institutions should work on ways to encourage 
quality reports and not let people off the hook for ignoring their obligations. 


#1. The “Burnout your best people” problem 

And finally, how do we reward people who go above and beyond or who always seem 
to be doing this work? Institutions should look for ways, even simple ways, to reward 
best practice. Some institutions actually have small monetary rewards for programs that 
complete the best reports, and these modest funds can be used for equipment, research, 
travel, etc. Other institutions may invite their most engaged faculty and staff to assessment 
or accreditation conferences—this serves as a reward while building institutional capacity. 
Finally, even a small gesture such as thank you from the President, a free lunch, or a small 
token of appreciation can go a long way. Every institution has that top 10% of people 
who are always doing their best in teaching, research, service, advising, assessment, and 
other tasks—your final question and discussion should be about creating a system that 
acknowledges and rewards these people. 


Wrap-Up 


After you have a process in place, it may be helpful to reflect on what problems still exist 
on your campus and what you can do to assist in removing these issues (and of course 
you may encounter even more challenges). This white paper provided an overview of 
best practices in program reviews and the importance of including assessment in these 
reports. The Appendices should be useful as starting points or comparisons. The following 
questions and discussion questions should help to guide the process. Best wishes on your 
continuous improvement journey. 


Question and Discussion #1: What are we even talking about? Developing 
shared definitions. 


Question and Discussion #2: What do your accreditation organizations have 
to say? Aligning with external accountability requirements. 


Question and Discussion #3: What will work for your campus: The why, 
when, how, who, and what of program review? 


Question and Discussion #4: So What? How will we “Close the Loop” and 
engage in Continuous Improvement? 


Question and Discussion #5: Forewarned is forearmed: What are the potential 
problems? 
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